Text-to-Image Generation based on Crossmodal Association with Hierarchical Hypergraphs

In this paper, we propose a novel framework for text-to-image generation based on association between text and image modalities. As an association model, we use hierarchical hypergraphs which consist of two layers including heterogeneous hypergraphs. While the first layer is composed of two hypergraphs: a text hypergraph and an image hypergraph, a hypergraph in the second layer associates two modalities by merging two hypergraphs in the first layer. In our model, hypergraphs are learned by self-organizing method based on random sampling. With multimodal association represented in the learned model, an intermediate image is generated by cross-modal inference when text keywords are given as a query. We use Korean magazine articles as a text-image data for experiments and we illustrate generated intermediate images and retrieved images similar to the intermediates as experimental results.